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ABSTRACT 

The effects of monetary gratuities on response rates 
to mail surveys have been considered in a number of studies. This 
meta-analysis examined; (1) the nature of the population surveyed; 
(2) the effects of gratuities in relation to the number of 
follow-ups; (3) whether the gratuity was equally effective across 
differer.*; populations; (4) whether the gratuity was promised or 
enclosed; and (5) the year of publication of the study. The bulk of 
the studies was done in the context of market research. Findings 
indicate that a monetary gratuity can be a powerful tool for 
decreasing non-response bias in mail surveys. Although follow-up 
mailings generally increased response rates in themselves, the 
gratuity typically had more impact than did an additional mailing. 
Thf- gratuity was almost equally effective for professional and 
general populations. Enclosing the gratuity was more effective than 
was promising it upon return of the questionnaire. The year of 
publication had some effect i*^ that the impact of the gratuity was 
slightly greater in earlier studies. The additional expense of the 
gratuity to improve response rates appeared cost-effective for most 
purposes. These findings have clear implications for survey practice. 
When differences in response with and without gratuities were found, 
they tended to show greater response integrity in the 
gratuity-receiving group. (SLD) 
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The mailed questionnaire is the most common type of data-gathering procedure 
employed in survey research. Despite its widespread use in the social and behavioral 
sciences, the proportion of the surveyed sample who elect to not participate is usually 
substantial and questions of non-response bias present serious inferential problems. With a 
low return rate, there is no entirely satisfactory answer to tne question, "Is the 
experimentally accessible population representative of the target population?" Bias checks 
can be very useful , but there is no substitute for maximizing the proportion of respondents. 

Several studies over the past half-century have found substantial increases in 
response rate fand conscouent external validity) from the use of monetary gratuities (Kanuk 
& Berenson, i975; Yu & Cooper, 1983). Most of these studies, however, have been done in 
the context of market research where general populations have been surveyed. The extent 
to which these findings are generalizable to content domains in which the respondents have 
a o-eater mtnnsic interest is uncertain. Previous reviews have either ignored or not fully 
utilized the power of meta-anafysis. 

The current study considers the nature of population surveyed, the effects in 
relation to the number of follow-ups (if effects are minimized or disappear when follow-ups 
are used), whether the gratuity is equally effective across differing populations, and if the 

gatuity IS promised or enclosed. The year of publication is also considered to see if the 
idmgs are consistent over the past several decades. 

The reviews of the effects of monetary incentives have not been entirely 
Sx^*'?? f^c^^rlein & Baumgartner, 1978; Yu & Cooper, 1983; Fox, Crask & Kim, 
1988). i\Iso, questions of cost-effectiveness and differential effects depending on the 
^?2P""^ °^ gratuity, and whether follow-up mailings are utilized continue to persist The 
diffenng conclusions may be due in part to three shortcomings in current reviews. The 
earlier reviews based their conclusions upon c small number of studies from limited data 
bases (e.g., the Heberlein etal review used only 18 studies; the Fox et al review was based 
on only 15 studies). In addition, monetary incentives have often been treated as a single 
vanable, regardless of whether the incentive was given as a gratuity prior to the return of the 
questionnaire or whether it waspromiscd after the questionnaire was completed and 
returned. If the two "modes of delivery" have differential effects, the failure to distinguish 
between the two strategies confounds the ef -cts on return rates. 

Previous reviews have also failed to accc uiit for the changing v{»iue of the dollar. 
The psychological and purchasing-power value oi a quarter incentive in 1931 when the first 
study was published is obvic mly quite different fiom the same coin today. If the amount of 
the mcentive at the time the study is net calibrated to a common metric, the magnitude of 

♦Paper presented at :he annual mating of the .'American Educational Research Association, San Francisco, 
March, 1989. 
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the effects for various amounts of gratuities will be distorted. In this studv the U.S. 
Consumer Price Index was used to calibrate the value of the monetary incentives for the 
time-frame of each study to 1987 dollars. 



Method 



The review of literature began with an on-line computer search of data bases in 
education, sociology, and business. Reference lists in the studies captured in these searches 
were then examined to identify additional pertinent studies. More than thirty studies from 
several different disciplines were located which inch' led both an experimental (£ or 
gratuity) group and a (usually randomly assigned) control group (C c • no gratuity). These 
studies provided more than fifty effects for the meta-anafysis from wn.ch the influence of a 
gratuity on response rate could be examined. 

The coding for the meta-analysis considered several factors: ( 1) the amount of 
55f ^^'I!^' calibrated to 1987 doUars, (2) whether the fatuity was enclosed or promised, 
(3) whether the population surveyed was general or professional,* (4) whether or not foUow- 
ups were included, and if so how many, (5) the increase in return rate for each foUow-up, (6) 
the year the study was conducted (or published if inf jrmation was not given), and (7) 
sample size. Two effect-size outcome measures^ for the gratuity were calculated from the 
response-rate data: (1) the absolute increase (A) in the percent of the sample returning the 
survey instrument (the difference in percent of response for the respective experimental and 
control groups), and (2) tht relative increase (R) in response rate (the ratio ofthe 
experimental ^oup s response rate to the response rate ofthe control group). Because of 
its lack of ambiguity, the findings will focus on the absolute difference except when the 
pattern of effects is different for ^ and R. Of special interest is whether the effect of ^he 
monetaiy gratuity depends on the response rate under control conditions, e.g., is the effect 
of the gratuity mfluenced by the willingness of the C group to respond without a gratuity? 

Analyses were conducted on each of the coded factors individuaUy. In addition, 
the collectwe impact that the respective independent variables on the response rate was 
exammed via multiple regression, which optimized the prediction ofthe return rate using 
the vanous coded variables as predictors. 

Results 

Taken as a ffoup, the mean response rate for the initial mailing in the C groups 
^2-^^^ = 33.3%); the variability, as expected, was substantial: 
s = 13.2%, with a range from ?0% to 59%. The fact that the bulk ofthe studies were done 
m me context of market research (where the average response rate is about 40% (Heberlein 
& Baumgartner, 1978) probably accounts for the generally low return rates. 

. Descriptively [see box-and-whiskerplots in Figure 1] the distribution was slightly 
positively skewed (7, = .27 and platykurtic (7, = -.76) (viewed inferentialhr, these values are 
not sufBcien* to reject the normaUty hypothesis, z. = .73; z, - 1.39). On the other hand, 
the response rate for the corresponding, £groups was "mucif higher (/ = 12.6, p < .0001), 
with a nean of 51.2% {Mdn = 50.0%)? T^e variability in return rates within the E group 

'Initial^ this &«or had three levels, but since the return rates for th* middle and low groups were virtually 
id|Bntical, the middle and low levels were combined 

^ince the outcome measure is not arbitrary, and is directly interpretable, it was judged to be a superior measure 
to the commonly used effect size (standard deviation) metric. 

^When a study did net provide separate return rates for the first and second mailings, in this instance, the total 
return rate was used for mailing #1; this causes a conservative estimate of the effect of the gratuity. 



3 



was somewhat greater: j-- 16.6 (t = 3.53,p < .001; Glass & Hopkins, 1984, p. 268), the 
range extended from 18% to 86%; greater variability in the gratuity group is not surprising 
since the amount of the gratuity, and whether it was enclosed or promised, also varied within 
the group. The distribution for^the 54 E observations differed little from a normal 
distribution (7, = .08, = .23; % « -.43, z, « .67). Not surprisingly, there was a high 
correlation between the £ and C response rates (r = .774; see Figure 1). 

Rgurc 1. Response Rates with (upper curve) and without a Monetary Gratuity (lower curve), with 

corresponding Box-and- Whisker Plots for E and C Groups, and for the Absolute Differences (A) in 
Response Rates. 
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Relationship between Absolute (A) and Relative (R) Return Rates. In order to 
determine whether the focus of the analysis could be on the absolute, rather than the 
relative effect of the gratuity, the relationship between absolute (A), and relative (R) 
measures was studied to determine whether the two variables yielded the same pattern of 
findings. The linear correlation between/l and R was substantial, .689. When the logarithm 
Qogm) transformation of R was used, the relationships with^ became essentially linear, and 
the linear correlation increased to . 760. 

. . , The correlations of the absolute and relative effects with the other coded 
vanables were similar with two logical exceptions: A was positively correlated with the E 
return rate (.62), and R was substantially negatively correlated (-.54) with the C return rate. 

AINU ^® absolute effect of the gratuity (A) did not depend the C return rate (r = - 
.01); thus, the absolute incre£ae in return rate was stable regardless of the return rate under 
normal conditions. In other words, the effect of the fatuity was to increase the absolute 
response rate (A) substantially~by an average of 18.0% (Afdn = 17.0%), regardless of 
whether the base retuir. rate was low or high. The distribution ofyl [see Figure 11 was quite 
variable (j = 11%) and skewed positively (7, = .58). The .95 confidence interval for the 
mean effect on the absolute response rate extends fi-om 15.1% to 20.9%.* 

A more complete graphic display of the ^. . ings is given in Figure 2. The lower 
curve gives the response rates for each of the control groups ^ank-ordered from low to high); 
the upper curve gives the response rates for the associated £ groups. Notice that in 53 of 
the 54 instances, the return rates associated with the E groups were higher. fThe single 
exception is fi-om Denton (1988), a study with a very small sample size (36)].' 

The Relationship between Ancillary Factors and Effect 

Promised vs. Enclosed. Enclosing a monetary gratuity appears to have a much 
more powerful effect than when a gratu:*?y is promised upon the return of the questionnaire. 
Enclosing a gratuity increased the absolute response rate by almost three times as much as 
did tlie promised gratuity, 20.0 (n = 45) vs. 7.7% (n = 8)! This occurred in spite o*' 'he fact 
that the promised gratuities tended to be larger. Indeed, when ANCOVA was used to 
equate the two groups on amount of gratuity (log,^ of 1987 dollars) and type of population, 
the adjusted return rate for the promised gratuity differed very slightly fi-om that without a 
gratuity. Perhaps the promised gratuity is perceived more in the 'pa/' mode (in which case 
It is virtualty ahvays inadequate and could be viewed as an insult), rather than as a gratuity. 
Note also that since the overall absolute effect described above was an aggregate of both 
enclosed and promised gratuity, the effect of enclosed gratuities perse is underestimated; an 
absolute increase of 20% is a better estimate based onfy on those studies with enclosed 
monetary gratuities. 

Amount of Gratuity. Using the value of the gratuity (calibrated to 1987 dollars), 
four levels of the gratuity factor were formed: (1) $.50 or less (n = 17; mean 
gratuity = $.29), (2) $.51.$.99 (n = 13; mean = $.75), (3) $1..$1.99 (n = 10; mean = $1.30), 

*AU statistical inferences in this anafysis assume that the obsei vations (effects) are statistically independent. 
Since several studies contributed more than one effect (e.g., had two or more E groups with different gratuiiy 
ampunts, each being compared to a common control group), the effects are not purely independent. The studies 
contributing only a single effect were compared with studies having more than one effect, in both means (t) and 
variance (F) with respect to return rates for C, E, and A and R. In none of the eight inferential tests was 
agnificance approached, the smallest p-value being .15 for A (where the mean for the studies with a single effect 
was somewhat higher~21.2% vs. 16.6%). Consequently, it is judged thai the consequence of imperfect 
independence does n M seriously distort the inferential statements. 
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and (4) $2 or more (n = 14; mean $3.84). Since the groups differed with respect to 
promised-vs.^nclosed gratuity, it was necessary to covaiy on this factor. The ANCOVA 
aUowed the null hypothesis to be rejected for the absolute increase in response rate 

- ':^'P < ""l)- The effect of the gratuity in the lowest group (adjusted 
mean = 5%), was much below that of the three groups with larger gratuities. 

Tvp? ofPQpulfffipn. Under usual (C) conditions, the return rate for the 
professionalpopulations (38%, and coded "1") was slightly greater (r = .09) than for tKe 
\ general («|%, and coded "0") populations. Of special importance is whether the gratuity 
effect interacts with type of population; the effect of the gratuity was almost as strong within 
professional groups as for the general population--the correlation between population and 
absolute increase in return rates was only -.08. This finding is consistent vdth that reported 
byGoodstadtc/fli (1977). ^ 

u °^ population factor is prooably related to the salience of the content of 

the survey, which has been shown to be highly related to return rates (Heberlein & 
Baumgartner, 1978), thus the issue is more complex than just the definition of tht target 
population. * 

Effftff gn4 FqUqw-up^s. Only nine studies included more than one maiiwg. 
Follow-ups were less common in the earlier studies; they were also more common vAth 
professional populations. Only three of the studies included two follow-ups. 

It is well known that follow-up mailings substantially increase return rates. What 
i3 of particular interest in the present investigation is whether the effect of gratuity is 
reduced or eliminated by additional mailings. In the nine studies having more than one 
niailmg, the response rate for the control group increased from 35.5% to 54.3% (18.8%) 
after the second mailing-veiy similar to the figure of 19.9% re'^oned in Hebcrlem & 
Baumgartner s 0978} review.. In the experimental group the fnitial return rate of 58.7% 
mcre^ed to 68.0% (9.7%) after the second mailing. The absolute difference in response 
i^'SIr ^ ^ groups in these nine studies decreased from 23.2% after one mailing to 
13.7% after two mailings; the gr tuity effect remained substantial even after two mailings. 
Note that the return rate fc < jup C after the second mailing remained below that of the E 
group for only one mailir'- 

* having three mailings, an absolute difference in return rates 

of 10.4% fevonng the E group remained even after the third mailing. Note that the return 
rate in the E group after two mailings (82.4%) remained higher than that for the C group 
after three maihn^ (78.7%). Thus, it can be concluded that the effect of the gratuity on 
return rates remams large enough to be of practical significance even when three mailines 
are involved. " 

IVffr QfPH^lmti(^4. The 54 effects came from studies having a mean year of 
publication of 1970 (Mdn = 1972), with a range from 1931 to 1986. Year-of-publication did 
not cx^aelate with adjusted amount of gratuity (r = .01) or the promised-vs.-enclosed 
dichotomy (r = -.00), but did have a slight correlation with the return rates (r = .19), 
perhaps suggesting some methodological progress in return rates over the last half-centuiy 
(the correlations remained essentially unchanged when the type-of-population was 
partialled out). The impact of the erataity appears to have been slightly greater in earlier 
studies; the correlation of Year with the absolute difference was -22 (p =.06). 
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Predicting Return Rates. 

To assess how accurately the total (percent) response rate (9) could be predicted, 
and to assess the relative contributions of each of four independent variables in the 
prediction, a multiple regression equation was developed based only on those studies with 
enclosed gratuities using the following predictors: 

G: Gratuity group (0 = None, I £ $.50, 2 = $.51-.99, 3 = $1-1.99, 4 > $2) 

M: The number of mailings (7, 2, or 3) 

P: Type of Population smvej«d(7 = General, 2 = Professional) 

D: Decade of the stu(jg jyBel5fe^ 50, 5 = 1950s, ...,8= 1980s) 

A multiple correlation of .627 resulted (.600 after correction for error-fitting), 
usin^ the raw-score regressions equation below.) Beta weights are given '^elow each 
predictor) to help assess the strength of each predictor.) 

f= 7.47G-¥ 12.23M ■¥ 3.68P .89D + 13.2 
.532 .226 .096 .050 

For example, the predicted rate of return for a mail survey of a professional 
population with a $1 gratuity enclosed and two mailings m the 19"0s is: 

^ = 7.47(3) + 12.23(2) + 3.68(1) + .89(8) + 13.27 = 70.9%. 

The same s' \dy with two mailings, but without any gratuity would have a 
predicted return rate c.'only 48.5%; the increase in return rate from the $1 gratuity is 
predicted to be 22.4%. The standard error of estimate for these prediction w quite large 
•14.9%); the predictions are far from precise. As would be expected, there are many other 
factors that influence return rates than those explored here. 

Cost/Benefit Considerations 

Using the predicted return rates from the regression equation above, and 
assuming a professional population survey by mail under non-gratuinr conditions, the return 
rate for the mitial mailing (m the 198Cs) is estimated to be only 36.4%; as indicated above, 
this would be predicted to mcrea^e to 48.5% after a second mailing. Assuming first class 
mailings of one ounce or less, with enclosed stamped return envelopes, the cost per returned 
questionnaire would average approximately $1.37 for a single mailing, and $1.6isrfor two 
mailings. 

The same Dopulation surveyed with a $ 1 gratuity, would h we predicted PJtum 
rates of 58.6% and 70.9% after one and two mailings, respectively. The mailing costs would 
average $2.56 per returned questionnaire after one mailing and $2.41 for two fxiailings (no 
gratuity enclosed in second mailing). If a 25c gratuity were enclosed, the return rates for 
one and two mailings would be predicted to be 43.7% and 55.9% respectively. The cost per 
returned questionnaire would average $1.72 for a single mailing, and $1.85 for two mailings- 
the latter value being less than 10% above that for the non-gratuity group. 

Of cotme these cost comparisons do not give the benefit perspective. The critical 
question is, ' docs the improved external validity of the statistical information justify the 
additional cost?" With two mailiiiBs without a gratuity, a return rate of only 48.5% is 
predicted, thus more than one-half of the target population is not being represented. With a 
$1 gratuity and two mailings, the predicted return rate of 70.9% is a great impro>'ement. 
Related to typical manpower costs, the additional expense to shore up a chief weakness of 



mail surveys would appear to be highly cost-effective for most purposes. The $400 addition 
cost of the prototypical survey « = 400 with a $1 gratuity appears nominal. 

Even in a fixed, hiinimal cost context, the gratuity strategy should be considered. 
A smaller sample with a gratuity 71% rate of return will be more credible than a larger 
^^^Pl^/OJ^^^ ^ "^^^ ^^^^"^ ^^^^ without the gratuity, it would take 206 mailings to 

yield 100 returns, whereas with the gratuity it would take only 141. Note also that the study 
with a larger sample also has adc«ition costs other than mailing costs-more envelopes, more 
copies of the questionnaires, more labor costs. All things considered, the survey of 300 with 
a $1 gratuity will cost less money than a survey of 600 without a gratuity. But the important 
methodological question remains, "which study will provide a more valid representation of 
th i terget population"? Stated differently, which study would yield a more believable 
confidence interval? Most statisticians would prefer findines based on data from a return 
rate of /l% for a sample of 300, than those based on data from sample of 800 with a 49% 
return rate because the larger sample has a 51% non-response rate vs. a 29% for the smaller 
sample. (Of course bias checks, e.g., comparing the responses from the first with the second 
roailmgs, remam cntical even with the higher response rates.) 

Another promising option in terms of cost-efficienQr that appears to have been 
essentially unexplored is the use of a gratuity only in the second mailing to those who did not 
respond initiaUy. The only study (Huck & Gleason, 1974) on this variation found the 
gratuity effect in the initial mailing of 28% was reduced to only 7% when a gratuity was 
included in the second mailing to the C (but not in the second mailing to the E group). 

Discussion 

Survey research findings based on a low response rate is likely to serve poorly as a 
basis for pohcy decisions and practice. Because the mailed survey is so frequently 
incorporated mto the research/decision making process by policy makers in government, 
education, and society, it is particularly important that the quality of the research data be as 
good as possible. 

The findings of this meta-analysis indicate that a monetary gratuity can be a 
powerful tool for decreasing non-response bias in mail surveys. The results also indicate 
that the use of additional follow-up mailings reduce the difference between the E and C 
return rates, additional mailings, although important, do not eliminate the practical benefit 
of the gratuity. The gratuity typically has more impact than an additional mailing. A 
monetary oratuity appears to be almost equally effective for professional (who typical would 
be assumed to have more intrinsic motivation to respond) and general populations. These 
findinK appear to have clear and strong implications for survey practice, especially in light 
of the fact that comparative studies have found no or negligible difference in the response as 
f «o r-^^fl"*'?^ °^ jgratuiw (Hancock, 1940; McDaniel, 1980; Maloney, 1954; Mizcs et al., 
1984; Shuttlcworth, 1931; Wotniba, 1966; Zusman & Duby, 1984). When differences have 
been found, the have tended to show greater response integrity in the gratuity group. 
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